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ABSTRACT 

REVIEW OF STATE VR AGENCY PROCEDURES FOR CASE FLAGGING: A QUALITY 
ASSURANCE ACTIVITY , by Susan Stoddard, Ph.D., J. MaTk Rogers, Shirley 
Langlois, Caleb Whittaker, and Deborah Kogan, Berkeley Planning Associates, 
April 1983, 51 pp. 

This report reviews the practices implemented by 53 different state 
VR agencies to address quality assurance concerns. After reviewing the 
broader spectrum of monitoring practices and evaluation systems used by 
different state agencies to assess worker and agency performance, the 
report focuses on the range of procedures xvtilized to track timeliness , 
which has been adopted as one of the procedural standards 'of service quality 
in the revised Standards Evaluation System. The report also reviews actual 
findings on the extent of timely versus untimely service* in four of RSA's six 
Model Evaluation Unit (MEU) states, based on data from a small client 
sample in each state. The report recommends that, rather than adopting 
absolute standards to define timely versus untimely completion of each 
status, states implement case flagging systems that would periodically 
identify cases to receive timeliness reviews. The report also suggests 
that states consider building adjustments into their timeliness flagging 
systems to account for variations in the expected time in status for 
clients with different characteristics. 
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I. INTRODUCTION 

A TYPOLOGY OF QUALITY ASSURANCE ACTIVITIES 

This report reviews the variety of practices currently utilized by 
state VR agencies to carry out quality assurance functions, in general, 
and to ensure the timeliness of VR services at the individual case level, 
in particular. In most agencies, the purpose of a quality assurance 
program is either to monitor the quality of the services that are being 
provided, or to evaluate the quality of the services that have been pro- 
vided. Monitoring implies close supervision of current operations; eval- 
uation implies assessment of past performance. While most comprehensive 
quality assurance programs contain elements of both monitoring and evalua- 
tion systems, it is usually possible to determine which of these is empha- 
sized in any program. For each of these types of quality assurance systems, 
the following aspects of agency design and implementation require attention: 

• the standards or criteria used to define the ideal; 

• the level within the VR agency's hierarchy at which the 
review is held; 

e the instrument used to assess performance; and 

• any other methods of assuring or evaluating service quality. 

To make this review of current quality assurance practices as repre- 
sentative as possible, Berkeley Planning Associates sent letters request- 
ing any materials related to quality assurance and case flagging activities 
to all VR agencies in the States and U.S. Territories. Of these 85 requests, 
53 agencies responded with a wide variety of manuals, review instruments, 
and other sources of information, all directly or indirectly related to 
quality assurance. We welcomed this variety of materials and have repro- 
duced representative materials from each state agency in an Appendix to 
this report (bound separately). While it might have been easier to sum- 
marize the responses had we requested specific information or conducted 
a pre-coded survey, we did not want to limit the scope of the responses by 
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imposing a predetermined definition of "quality assurance" on the state 
agency respondents- It was important to us to understand what a particular 
agency thought of when we requested "quality assurance" information. Using 
this method allowed for an indication of an agency's emphases, and yielded 
some interesting results. 

Figure 1 demonstrates the variety of methods employed by states to 
control for quality and timeliness in their respective case management 
systems. The names of states responding to our request to send materials 
(describing their quality assurance and case flagging procedures) are 
listed in the first column. Also in this column is a code letter and num- 
ber for easily identifying the corresponding materials included in the 
Appendix. 

Figure 1 includes four columns under the heading "CASE FLAGGING," 
noting the different criteria used: Norm for Maximum Time in Status; 
Time Since Last Contact; Time in Status (no specific maximum); and Routine 
Review Schedule. An "x" in any of these columns indicates that the par- 
ticular state in question uses that particular criterion for flagging 
ccses. Because states differed in their practice of flagging cases in 
all statuses or some statuses, this information is included in parentheses 
iYi the first column. 

The last two columns on Figure 1 attempt to classify each state's 
"QUALITY ASSURANCE APPROACH" as either "Monitoring" or "Evaluation," or 
both. Descriptive phrases are included in parentheses in an effort to 
acknowledge the variation in types of monitoring systems: in-house; use 
of SDCRS (San Diego's Case Review Schedule); informal; etc. Likewise, 
the final column, "Evaluation," includes frequency or scope of the eval- 
uation in parentheses. 

QUALITY ASSURANCE THROUGH MONITORING PRACTICES 

Most of the agencies that responded to our request for information 
have quality assurance programs primarily devoted to monitoring current 
activities. One of the advantages of a monitoring program is that it 
allows for a quick identification of potential problems before they have 
a chance to become entrenched. By correcting these problems early, the 
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monitoring program can improve the quality of services currently being 
provided to clients in the syscem. This minimizes the "after the fact" 
nature of problem-solving. However, one of the disadvantages of a monitor- 
ing program is that monitoring activities usually occur at the caseworker/ 
supervisor level, and thus a clear picture of the agency as a whole is 
difficult to develop. 

Quality Assurance Standards in Monitoring 

The logical place to begin a discussion of quality control procedures 
in monitoring programs is to review those ideas or concepts that agencies 
consider important enough to elevate to the level of "standards." These 
standards represent the ideal toward which the quality control process is 
moving. Standards can be grouped into broad areas; they can be generated 
in-house, or can be part of an established quality assurance program in 
place in more than one agency. Once this determination has been made, 
standards can be further grouped into those that are process-related and 
those that are outcome-related. 

In the respondent agencies with quality assurance programs primarily 
concerned with monitoring current activities, most of the standards used 
for measuring performance were generated in-house (although a few of these 
are based on federal requirements). By examining the similarities in these 
in-house standards, the concepts or ideas that agencies consider to be 
particularly important to their respective operations are readily apparent. 

Most of these in-house standards used by monitoring programs are 
process-related . This type of standard emphasizes proper casework practice, 
especially in the area of documenting client services. The areas where 
proper procedure is considered to be particularly important include: 

• determining eligibility; 

• placing a client into extended evaluation; 

• closing a client as ineligible for services; 

• completing the IMP; and 

9 closing a client as rehabilitated or not rehabilitated. 

Timeliness is one key procedural standard that can be applied to 
each of these program transitions. An interesting observation is that 
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few of the in-house standards used in the respondent states in monitoring 
their own programs are outcome-related . Even though these programs empha- 
size monitoring activities, one would expect a.t least a few outcome-related 
standards to ensure that the monitoring standards are ultimately related 
to program effectiveness. The few monitoring programs that include outcome- 
related standards as part of their quality assurance program are mostly 
from the larger agencies. This might mean that smaller agencies simply 
do not have the resources available to develop and measure performance on 
outcome-related standards. If resource availability is a problem in the 
smaller agencies, then perhaps these agencies should consider adopting some 
of the available outcome-related standards packages to supplement their 
process-related standards . 

Level of Review in Monitoring Activities 

A very important aspect of any quality assurance program is the level 
within the VR agency structure where quality assurance activities apply. 
In most VR agencies, monitoring activities are held at a relatively low 
level, usually starting at the counselor/supervisor level. The activities 
are held at a certain time interval (usually monthly), at status changes, 
or at critical points in the service process (acceptance, IWRP, closure). 
In most of these cases, the counselor's performance is reviewed using' 
process-related standards- As monitoring activities are usually conducted 
at the counselor level, it makes sense for monitoring programs to emphasize 
process over outcome. Oftentimes, these lower-level monitoring activities 
move upwards; for example, the counselor is reviewed by his/her supervisor, 
who is in turn reviewed by his/her superior, and so on. 

The basic purpose of a quality assurance program is to identify 
problems and then correct them. Only rarely did our respondents describe 
the process by which problems identified through quality assurance activ- 
ities are corrected- In those cases where a "feedback" process was des- 
cribed, it usually consisted of the reviewer and the reviewee negotiat- 
ing an "action plan." This "action plan" is followed by a re-examination 
at a later date. 
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Case Review Instruments Used in Monitoring 

By far the most popular method of implementing quality control in 
both monitoring and evaluation systems is the case review. Almost with- 
out exception, the responding agencies gave an indication that case reviews 
compose a major part of their quality assurance programs. This is one 
are; in which monitoring and evaluation systems show little difference. 
What differs among agencies and across the monitoring and evaluation 
functions is the length of the instrument and its focus. 

The majority of state agencies that responded to our information 
request design their own in-house case review instruments for use in 
monitoring. These instruments range from just a few items to a complex, 
detailed case review instrument. But, for the most part, the case review 
instruments utilized in monitoring seek to ensure that proper casework 
procedures are being followed. As would be expected, these areas of empha- 
sis in case review correspond to the most frequently mentioned standards 
for monitoring the quality of services (most of these in-house instruments 
ask questions answerable by a simple "yes" or "no"). In a few cases, 
space is left at the end of the instrument for a brief narrative describ- 
ing the results of the review. As used in a monitoring system, case reviews 
also provide the basis for counselor performance reviews. As we will see 
in the following description of VR agencies 1 evaluation systems, case 
reviews may also provide information for broader agency-level performance 
reviews. In a monitoring system, however, case reviews and counselor 
reviews are part of the same process. 

A few agencies reported that they had found packaged case review 
instruments particularly suited to their monitoring needs. The most fre- 
quently mentioned case review instruments were the San Diego State Case 
Review Schedule (SDCRS) and the Assumption College Case Review. The San 
Diego instrument allows for review of the client's case record to ensure 
that all procedures were followed when a client was in any particular 
status, while the Assumption College Case Review emphasizes proper pro- 
cedure in advancing a client from one status to another. 
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QUALITY ASSURANCE THROUGH EVALUATION PRACTICES 

Evaluation systems differ from monitoring systems in that evalua- 
tion systems do not attempt to alter or improve current practice, but 
rather assess past performance in the hope of improving future performance. 
One of the advantages of an evaluation system is that it is more central- 
ized, and results are easily aggregated for the entire agency in a short 
period of time. A disadvantage is that evaluation systems do not improve 
services to clients currently in the program; they look instead to improve 
services in the future. 

Quality Assurance Standards in Evaluation Systems 

While monitoring systems emphasize process-related standards over 
outcome-related standards, the evaluation systems described by our state 
agency respondents usually combined elements of both. In our sample, 
most evaluation systems used standards developed in-house, which com- 
bined process-related and out come -related performance measures. The 
process-related standards are similar to those described earlier in mon- 
itoring systems. Of the outcome-related standards, most emphasize such 
things as : 

e comparison of the current year's performance to the previous 
year 1 s ; 

• retention of benefits; 
o client satisfaction; 

• client earnings information; and 

• job placement (competitive and non-competitive employment). 

There are also packaged standards systems currently in use in state 
agencies 1 evaluation systems. Among those mentioned by states are the 
BPA Program Standards and the Georgia Management Control Project Standards. 
Both of these systems received high marks from their users. The BPA Pro- 
gram Standards concentrate on outcomes and impacts, especially in terms 
of agency efficiency and client services. 1 The Georgia Management Control 



Berkeley Planning Associates, Vocational Rehabilitation Program 
Standards Evaluation System; Using the System: An Analytic Paradigm for 
Management, June 1982. 
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Project acts as the opposite of a monitoring system; it seeks to reduce 
the constant monitoring of counselors by including counselors in. the objec- 
tive planning process and eliminating "superficial controls." 1 

In addition to comparing agency performance to previously established 
standards, many evaluation systems regularly compute and publish descrip- 
tive statistics such as the rehabilitation rate> acceptance rate, referral 
rate, referral backlog, and the ratio of closed to open cases for a specific 
period. 

Level of Review in Evaluation Systems 

While most quality assurance activities in monitoring systems take 
place at lower levels in the VR agency's structure, evaluation activities 
usually take place at a higher level.. In many agencies, a separate staff 
is maintained with responsibility for evaluation quality assurance. 

In those agencies with a separate quality assurance staff, the staff 
usually conducts a general review not focused on any specific counselor 
or client; that i>, the review operates more as an overall agency evalua- 
tion than as a counselor evaluation or a monitoring function. In the 
evaluation systems described by the state agencies, the reviews are also 
held less frequently than in monitoring systems (usually quarterly or 
semi-annually), and cover a larger sample of cases. 

An interesting observation is that very few agencies with evaluation 
systems discussed any quality assurance activities at lower levels such 
as the counselor /supervisor level. While this does not mean that no quality 
assurance activities are held at this level, it could point to an area that 
needs more attention. If an agency expects the quality assurance staff 
to detect all important problems in service delivery, it might be benefi- 
cial for them to re-evaluate the utility of assessing service quality for 
more disaggregated levels (e.g., by unit or by counselor). 

Another observation, which was noted earlier in the description of 
monitoring systems, is that few agencies described their processes for 
improving performance that is found to be problematic. This is especially 
true for evaluation systems. In those few cases where feedback mechanisms 

2 The Management Control Project: A Description , Georgia Rehabilita- 
tion Agency, 1980, p. VII. 
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were described, they usually consisted of staff meetings, reports, memoranda, 
and large-scale "action plans. ff It is possible that the greater "distance 11 
between the reviewer and reviewee at higher levels hinders feedback. (This 
is especially true if it is the "agency" that is being reviewed.) If this 
is the case, then agencies may need to concentrate on quickly disseminating 
review results and adopting. and promulgating action plans. 

Case Review Instrument 

As was noted earlier, there is little difference between monitoring 
and evaluation systems in terms of how case reviews are conducted. 
Although they comprise a major part of both systems, case review findings 
are emphasized less in evaluation systems than in monitoring systems, 
since they are designed primarily to check compliance with established 
procedures or procedural standards. 

In an evaluation system, case reviews are designed to supplement the 
use of outcome-related standards. However, agencies maintaining a separate 
quality assurance staff may have more resources available to invest in 
case review (in terms of staff time available and staff expertise) and 
thus may be able to conduct more thorough case reviews. \ 

\ 

Performance Assessments \ 

Since in an evaluation system the case review process is part of an 
overall agency review, some supplementary method of evaluating the per- 
formance of individual counselors must be developed. Only a few of the 
state responses describing their evaluation systems mentioned counselor 
performance reviews. In these cases, counselor performance was reviewed 
by means of smaller-scale case reviews, or by means of client satisfac- 
tion questionnaires. In a few cases, counselors wer^ previously given 
the specific standards or goals against which performance would be rated. 
The standards include such things as comparisons to the previous year's 
performance, and negotiated goals. For the most part, the performance 
objectives referred to the following aspects of program functioning: 

• referral and intake; 

• caseload mix; and 

• the proportion of 26 closures and competitive employment outcomes. 



9 

ERLC 



15 



10 



SUMMARY 

Most quality assurance systems can be grouped according to their 
relative emphases on monitoring versus evaluation functions. While some 
state programs have incorporated elements of both types of quality assur- 
ance, the program usually favors one over the other. In our sample, 26 
agencies had adopted a monitoring approach to quality control, two used 
evaluation, and 15 had elements of both. 

Monitoring systems seek to closely watch current practices with the 
intent of invervening to change poor practice as soon as possible. The 
following presents a summary of the observed similarities of the monitor- 
ing systems in the different states: 

• Monitoring activities take place at a lower level than do eval- 
uation systems, usually at the counselor /supervisor level. 

• Monitoring activities also take place more frequently than 
evaluation efforts (usually monthly). 

9 Most monitoring standards are process-related. 

• Monitoring is conducted by means of case reviews, using in- 
house assessment instruments. 

• Monitoring systems are closely related to assessments of case- 
worker performance. 

Evaluation systems , rather than monitoring current activities, assess 
past agency performance in order to learn how future, performance might be 
improved. The following is a summary of the observed similarities among 
the evaluation systems in the different states: 

• Evaluation activities usually emphasize agency -wide (or area- 
wide assessments. 

o Evaluation activities take place less frequently than do monitor- 
ing activities, usually quarterly or semi-annually. 

• Evaluation standards are mostly outcome-related, but can be 
process-related as well. 

• In evaluation activities, case review findings are supplemented 
by computed statistics, and comparison of program performance 
to outcome standards and other baseline data. 

• Evaluation systems are less well adapted for assessing the per- 
formance of individual caseworkers. 
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The variety of methods used across the different state agencies 
indicate that "quality" and "quality assurance" mean different things in 
various state agencies. BPA recommends that states consider an approach 
that includes both evaluation of agency outcomes as well as concern for 
compliance with program regulations and other procedural "quality assurance" 
standards. While it is important that state agencies follow program regu- 
lations and specified modes of operation, procedural compliance itself 
does not guarantee service quality. In fact, we would suggest that if the 
agency's performance goals as measured by clients 1 ability to obtain and 
keep jobs in competitive or other suitable employment are not met, then 
other measures of quality may be meaningless. This dual approach is 
reflected in the system of RSA performance measures, refined by Berkeley 
Planning Associates, which address concerns about procedural quality as 
well as performance outcomes.* 

Many agencies that submitted data for our review indicated that the 
San Diego Case Review, or another case review instrument, adequately served 
their compliance assurance naeds. So long as the state-federal program 
is based on federal regulations, this procedure at least ensures examina- 
tion of federal regulatory requirements. However, the relationships of 
these procedural standards to program outcomes has not yet been fully 
explored. We recommend rigorous investigation of these relationships 
before standards based on regulatory procedures are adopted as part of 
the quality assurance system for the national program. 



For an overview of the standards evaluation system, see Berkeley 
Planning Associates, Vocational Rehabilitation Program Standards Evalua- 
tion System: Executive Summary, June 1982. 
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II. TIMELINESS AS A QUALITY ASSURANCE CONCERN 



In Berkeley Planning Associates 1 pretest and refinement of the Voca- 
tional Rehabilitation Program Evaluation Standards,* an investigation of 
the appropriateness of including performance standards for case timeli- 
ness was carried out. In addition to the quality assurance material col- 
lected from 51 different state agencies, BPA developed information on 

case timeliness for a sample of clients in four Model Evaluation Unit 
2 

pretest states. The review in the following pages summarizes our find- 
ings and results in recommendations for the inclusion of timeliness as 

a procedural standard in the Standards Evaluation System. 
« 

The "Timeliness 11 Procedural Standard (#11) included in the Standards 
Evaluation System as refined by Berkeley Planning Associates seeks to 
avoid delays in the VR process that are likely to impede or hinder suc- 
cessful rehabilitation of the client. Rather than set a performance 
standard using time-in-status to define "undue delay, 11 this Procedural 



"Standard requires that ench state have a monitoring or flagging mechanism 
for cases remaining in statuses over a given length of time, and a pro- 
cedure to evaluate the appropriateness of any case delay. Many of the 
state VR agencies already have variations of such a system in place. 



services, for two reasons. First, one aspect of the quality of a client's 
service experience is the speed with which his or her case is handled: 
did the client feel that the counselor "cared 11 about him (as evidenced by 
the fact that the counselor "kept on top of things" and "kept things mov- 
ing along"), or did the counselor seem to put him on a lower priority? 
The client's perception of his treatment by VR can have an impact on his 



Described in Vocational Rehabilitation Program Standards Evaluation 
System: Final Report , "Executive Summary," June 1982. 

2 Because of the small size and nonrandom nature of the MEU samples, the 
findings on timeliness in the four state agencies must be considered prelim- 
inary, at best. Nevertheless, the findings do suggest how case flagging 
systems might be improved. 




This standard pertains to the RSA goal of providing quality case 
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attitude toward VR and about the usefulness of participation in VR. Second, 
research on successful rehabilitation outcomes has suggested a relation- 
ship between timeliness and success (perhaps as a consequence of the per- 
ceptions discussed above) . 

the issue of timely case movement or "undue delays" (as it is phrased 
in the current standards) has been one of long discussion and controversy. 
While there is literature to support the correspondence between certain 
times in process (particularly time to eligibility decision) and outcome, 
there have also been questions about interrater reliability in the area 
of judging timeliness of case movement through case review. Nevertheless, 
an overall review of timely case movement on a client -by-cli basis is 
best handled through case review, if items can be identified that have 
good interrater reliability. 

The issue of timely. case movement has generated much discussion, 
analysis, and controversy. Among the issues raised are such questions 
as the following: 

0 Is there evidence that clients actually experience undue 
delays during the VR process? 

• Do such delays actually impede or hinder successful outcomes? 

• Can we measure timeliness? Conversely, can we reliably 
identify occurrences of undue delay? 

In answer to the first question, it appears that some clients do ex- 
perience what appear to be undue delays. For example, previous reports 
of HEW Audits (1967-1974) indicated that clients were left in statuses 
often for excessive lengths of time, ranging up to over four years in 
some cases. * Reasons cited for such delays included: (1) lack of a 
monitoring system alerting counselors to abnormal lengths of time in 
status; (2) lack of contact between counselor and clients; and (3) dif- 
ficulties in establishing job objectives for the client. Other observers 
have noted that many apparent delays in service are, in fact, due to a 
failure to "move paper" on clients who are inappropriate for VR or unlikely 
Lo have successful outcomes, or to more widespread delays in clerical 
action by caseworkers who are "behind in their paperwork." The opinion 

1 HEW Audit Agency, Repor ts on the Administration of the Vocational 
Rehabilitation Program, 1967-1974. 
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was consistently expressed in the Audit reports that undue delays are 
detrimental to a client's potential for rehabilitation and that standards 
should be established for monitoring and following up on cases remaining 
in critical statuses too long. It should be noted that "too long 11 in a 
status is not specifically defined in the Audit reports, although some of 
the reports stated that six months is generally regarded as excessive for 
most statuses. 

Other researchers have also documented the existence of what appear 
to be excessive times in certain statuses. For example, the Region I and 
Region VI contributions to the "New Orleans Report 11 found that average 
times in certain statuses varied widely across state agencies 1 (e.g., large 
vs. small, general vs. specialized), and as a function of counselors ! s 
caseload types. 2 With respect to clients with differing. disabilities of 
differing severity, the expectation of what is an appropriate time in 
status may differ. After adjusting the expectation to fit clients 1 par- 
ticular characteristics, flagged cases that exceed the expected time 
in status can be reviewed to determine whether lack of movement represents 
"undue delay" for each particular case. In terms of individual clients, 
Region VI found longer times in status for the severely disabled and for 
males. A study by the Massachusetts Rehabilitation Commission found that 
there were some differences in the rate at which clients with different 
disabilities moved out of statuses 00-02 and 10, although no consistent 
pattern was noted. 3 Other variables examined in relation to time in 
status were demographic, community, and caseload variables. None of these 
studies, however, assessed whether such cases represented undue delay. 

The evidence is somewhat mixed as to the impact of undue delays on 
service outcomes. For example, there is some agreement in the literature 

1 Region VI (Cooper, P. and Greenwood, R.), "Assessing Undue Delays 
in the Vocational Rehabilitation Process," Studies on the Evaluation of 
State Vocational Rehabilitation Agency Programs: A Final Report, Stanford 
E. Rubin (ed.), Arkansas Rehabilitation Research and Training Center, 1975 

2 Region I, "The Relationship of Rehabilitation Counselor Caseload Size 
and Service Adequacy, Service Timeliness, and Caseload Balance," op* cit . 

Massachusetts Rehabilitation Commission (Johnson, B.D.), Analysis of 
Case Movement of Clients Between Application and Plan Development, 1974. 
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that favorable outcomes are facilitated by shortening time between referral/ 
evaluation (00-02) status and acceptance or initiation of services (status 
10) . A West Virginia R § T study revealed that the shorter the time spent 
in referral/evaluation statuses, the greater the chances of successful 
rehabilitation. 1 The Massachusetts Rehabilitation Commission study showed 
that fasver movement out of 00-02 is related to greater chance of accep- 
tance for services (status 10) and that faster movement out of status 10 
indicates more likelihood of successful rehabilitation. 2 However, a study 

by the Arkansas R § T Center found no consistent relationship between total 

3 

time in status and outcome. Urban Institute data show a statistically 

significant difference in times spent in referral-applicant status for 

cases closed 26, 28, and 30, although the direction of the relationships 

4 

is not consistent. Finally, a 1978 study by Berkeley Planning Associates 
found no statistically significant effects, on a variety of outcome measures, 
stemming either from elapsed time in the referral -acceptance phase (status 
00-10), or from time elapsed between referral and closure. 5 

Although the research to date does not demonstrate that successful 
VR outcomes depend on the timeliness, nevertheless, timeliness has been 
retained as an important measure of the quality of the service process. 
The importance attached to timely case movement is reflected by Standard 3 
of the 1973 RSA Evaluation Standards, which is intended M to ensure that 
undue delays are avoided in providing clients with VR services. H The 
standard sets upper limits on the length of time spent by clients in par- 
ticular phases of the VR process (i.e., 8 months for timely eligibility 

1 West Virginia Research and Training Center, "Looking at Referral - 
Applicant Statuses," Rehabilitation Tomorrow , Vol. 4, No. 3, March 1974. 

Massachusetts Rehabilitation Commission, op. cit . 

3 Cooper, P.G., Client-Counselor Contact, Undue Delay and Closure 
Status, Arkansas Rehabilitation Research and Training Center, 1975. 

4 Urban Institute, Final Report on Refinement and Expansio n of the 
General Standards for the Evaluation of the Performance on the Vocational 
Rehabilitation Program , 1976. 

5 Berkeley Planning Associates, Implementing the Reh abilitation Act 
of 1973: The VR Program Response, February 1978, 
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decisions and 22 months for timely completion of the VR process). By 
extension, clients who exceed these upper limits are presumed to be exper- 
iencing an "untimely" VR process. 

Given the policy importance attached to timely case movement, the 
question becomes "How can we measure timeliness?" Much effort has gone 
into attempts to define, and establish standards for, timeliness of case 
service progress. As noted, research on successful rehabilitation outcomes 
has supported the concern for timeliness. Previous attempts to monitor 
the timeliness of service provision by way of a standard on "undue delay" 
have been hampered, however, by validity and reliability problems. "Undue 
delay" and its converse, "expeditious" or "quick and efficient" case man- 
agement, mean different things to different people. The 1973 Standards 
approach, which sets normative time periods to define "timely" case movement, 
has been widely and justifiably criticized for its lack of sensitivity 
to the legitimate differences in individual cases: a complex case, 
perhaps involving long-term educational services, might well require 
more than 22 months without any delay. Where a case has been subject to 
a delay, the situation is further complicated by the differing implica- 
tions of different causes for delay: lack of client responsiveness, 
inattention or inefficiency on the part of the counselor or the VR agency, 
and problems outside of VR (failure of a vendor, to deliver, unavoidable 
waiting lists in training programs) . Each imply very different respon- 
sibility for time lapses and cannot equally be attributed to VR agency 
failure. 

Thus, use of objective or quantitative measures of timeliness has 
suffered from arbitrariness and frequent inappropriateness of established 
time cut-offs for many clients. Other approaches to objective measurement -- 
such as recording planned initiation and completion dates for each service, 
and monitoring compliance with the schedule— are cumbersome in execution. 
However, subjective or qualitative judgments of timeliness also have been 
vulnerable to criticisms. BPA believes, however, the major problem, un- 
reliability in application, may well arise due to the incorporation of 
two distinct concepts into raters 1 "undue delay" judgments. The term, 
"undue delay," includes concepts of both time lapse and judgment of blame, 
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culpability, or unjustifiable time lapse. (The word "delay 11 itself some- 
times connotes willfulness or negligence, and the modifier "undue" defin- 
itely implies such problems.) Case reviewers might well differ in judgments 
as to the cause of a delay, and thus whether VR should be held accountable. 
For this reason, reviewers may differ in their classification of a case, 
one citing an undue delay, another seeing an unfortunate time lapse, but 
being unwilling to label it an undue delay if client motivation or outside 
vendors played a role. 

Because of the problems associated with measuring timeliness, design- 
ing procedures to prevent undue delays becomes a complex administrative 
issue. On the one hand, states could "cast a wide net" to identify and 
address delays by lowering the standard for flagging and reviewing cases 
that have been in a particular status more than the "normative" (i.e., 
timely) period. However, such an approach would also increase the number 
of reviews and, thus, the total cost of case review activities. In the 
interest of efficiency, it is important to limit the number of cases 
flagged as being in status suspiciously long. However, by using the 
opposite approach (i.e., allowing particularly long amounts of time to 
pass before flagging cases), we maximize the potential for excessive 
delays to continue without corrective action. What is needed is a system 
in which cases are flagged after some "normal" period of time has elapsed. 
However, as illustrated by the above discussion, the term "normal" takes 
on different meanings, vis-a-vis time in process in different agencies, 
for different client groups, and for different service packages. Thus, 
an agency's case flagging system would ideally set the time cut-off points 
differentially for different statuses and for different kinds of clients. 
In this way, agencies would be able to arrive at a balance between the 
length of time allowed before flagging and the number of cases flagged 
for review. Also, this would provide the state with a functional tool 
for monitoring case movement while the client stands to benefit by the 
attention to his or her progress. The timeliness standards will not be 
periods of time that are required for every client. Nor will the flagging 
of a case mean that the case is necessarily mismanaged. Rather, the time 
periods should indicate the best available times to Teview the continued 
appropriateness and efficiency of the client's rehabilitation process. 
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In the remainder of this report, we discuss two topics related to 
service timeliness and case flagging. Fi-st, we present a "state-of-the- 
art" review of case flagging systems currently used by state agencies. 
Next, we present our analysis of the Model Evaluation Units 1 standards 
pretest data, obtained from the Timeliness Assessment Instrument, shown 
in Figure 2. This instrument attempts to overcome the measurement problems 
discussed earlier, by dividing case assessments to timeliness into two 
segments: first, a notation of whether undue delay has occurred in terms 
of a time lapse between successive phases in the case history; and, second, 
an assessment of the reasons for the lapses. The relevant questions con- 
cern critibal phases of case progress: (1) eligibility determination; 
(2) development of a service plan; (3) service delivery; and (4) termina- 
tion. Our intent was to identify factors (i.e., client characteristics 
and service aspects) that might be used to distinguish among clients when 
establishing flagging points, so as to improve the usefulness of a state's 
flagging system. At the conclusion, we present BPA's recommendation for 
a procedure that states may adopt, in conjunction with their own timeliness 
practices, to reduce undue delay. 
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Figure 2 

Information Items Obtained by the Timeliness 
Assessment Instrument for Reviewed Cases 



1. Was the case handled in a timely manner (i.e., without undue 
speed or undue delay)? 

2. If undue speed: 

a. Reasons for judging the case as moving too fast, 

3. If undue delay: 

a. Were the reasons for delay documented in the clients' case 
record? 

b. Reasons for delay • 



21 



III. REVIEW OF STATE VR AGENCY PROCEDURES 
USED TO IDENTIFY UNDUE DELAYS 

Many state VR agencies have taken the initiative in developing sys- 
tems for monitoring case progress. In this section we first discuss the 
range of timeliness monitoring methods used in the 48 state agencies that 
provided descriptive information to Berkeley Planning Associates. Next, 
we provide a more detailed discussion of the subset of systems that do 
include case flagging components. 

TYPES OF SYSTEMS USED TO ENCOURAGE TIMELY CASE SERVICE 

We identified three broad types of systems used in our responding 
agencies for assuring timely case movement. First is the group of agencies 
that provide some form of monthly client list with flags for delayed cases . 
Twenty-six of the forty-eight agencies have such systems, which are con- 
structed in various ways. For example, 13 of the responding agencies 
have established flagging times for almost every status (i.e., 00 through 
24). In these agencies, the "acceptable" times established for a given 
status vary over a wide range from status to status and from state to 
state. The remaining 13 agencies flag only particular statuses; usually, 
the "non-service" statuses such as 00-02, 06, 10-12, and 20-24. The desig- 
nation of "acceptable" times varies widely for this group as well. 

The second type of system used for timeliness assurance is that which 
provides monthly client lists without flags ; this system is used in 14 of 
the 48 responding agencies. Agencies using this system usually provide 
counselors and counselor supervisors with computer read-outs listing all 
active cases (i.e., cases in 00-06, and in 10-24). No time limits are 
established. Instead, these agencies rely on counselors and supervisors 
to monitor case progress, identify seemingly excessive times in status, 
and initiate case review and corrective action. This type of system may 
satisfy the monitoring needs of very small agencies, which may be able to 
adequately monitor each case's progress simply by scanning the total 
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client list. In contrast, agencies with large caseloads may have a 
greater need for a mechanism (e.g., a case flag) to highlight the cases 
making slow progress. 

Finally, eight of our responding agencies described systems that 
rely on individual case reviews . While such agencies may provide monthly 
client lists, they tend to rely on reviews of individual case files in 
order to identify undue delays. In some cases, case file reviews are 
conducted only as required by law (e.g., 90-day review during extended 
evaluation, annual reviews of the IWRP) . In a few cases, additional 
reviews may occur, as in agencies that use random sampling for case reviews 
as part of the agencies 1 overall quality assurance mechanisms. 

Each of the three systems described above have one common trait: they 
all provide some mechanism for intervening on behalf of a client while 
he or she is still on the case roll. In addition, we would like to make 
note of a system that is currently undergoing initial experimentation, 
and that uses a different approach to encourage timely case movement. 
This is the proposed weighted case closure methodology developed by the 
Virginia Department of Rehabilitation Services. In that agency's efforts 
to assure quality case handling, it is currently experimenting with a 
weighted case closure formula that incorporates a time factor into the 
calculation. 1 Without going into the details of the weighting formula, 
it is sufficient to note that the time component focuses on the time spent 
in particular statuses perceived as potentially problematic by the agency: 
status 02 (application); status 10 (plan development) ; status 20 (ready 
for employment); and status 24 (service interrupted). Using a statistical 
norm approach (i.e., deviation from average time elapsed) the methodology 
then assigns "points" to the case for overall weighting. Such a weighting 
system could be adapted as desired by individual agencies to reflect their 
particular priorities regarding statuses of interest. (For example, agencies 
could incorporate such time components as total time elapsed during eligi- 
bility determination (status 00-02/06), combined time in status 10 and 12, 
time spent in training (status 18), total time in service (status J'--.--.*; 

1 This system description was obtained from a presentation by the 
Virginia Model Evaluation Unit, at the VR Evaluation Conference in Scotts- 
dale, Arizona, in July 1981, 
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and so forth.) The point is that this system is intended to operate as 
an incentive mechanism in the overall evaluation of worker performance, 
rather than a monitoring/intervention mechanism on a case-specific level 
as is true for case flagging systems. Some questions remain about the 
usefulness of case weighting systems for encouraging timeliness. First, . 
as implied above, weighting systems will not assist agencies 1 intervention 
on current clients 1 behalf, since the weighting is done after closure. 
Thus, we would need assurance that the inclusion of a factor for timeli- 
ness in the formula actually served as an incentive to workers to improve 
timeliness. Second, we would need some evidence that the timeliness 
factor exerted a sufficient influence on counselors 1 behavior to overcome 
or at least balance the effects of other components in the weighting for- 
mula. For example, the Virginia system incorporates concerns for service 
to priority groups (i.e., the severely disabled), for quality outcomes 
(measured by closure wage), and for reduced service costs. The first two 
of these concerns could possibly serve to increase the time- in- service • 
averages. To the extent that counselors trade off timeliness for quality 
outcomes and/or service to the severely disabled, concern for timeliness 
is reduced. Finally, the weighting system is currently applied only to 
26 closures, although the agency sees the potential for extending it to 
28s. Until this is possible, weighting systems will fail to address one 
of the key issues that motivated the development of case flagging systems 
to avoid delays for those who may never achieve a successful outcome, as 
well as to reduce the total time in program for 26 closures. 

We do not propose that RSA or states adopt a case weighting system 
as a substitute for a case monitoring system, given the problems discussed 
above. However, if after Virginia's experiment, it appears that the time 
factor does influence counselor behavior, then consideration might be 
given to use of a weighting system as a complement to the state's monitor- 
ing system. 

In summary, about half (26) of our responding agencies use some sort 
of case flagging system. Fourteen of the agencies provide monthly client 
lists without flags, which are visually scanned to detect excessive times- 
in-status. Eight agencies rely on other methods -- primarily case review 
to detect occurrences of undue delays. The Appendix contains examples of 
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48 state systems. Below we present a more detailed discussion of the case 
flagging systems in use in state agencies. 

EXISTING CASE FLAGGING SYSTEMS 

The internal structures of case flagging systems vary considerably 
across state agencies. The dimensions over which these systems differ 
include: 

• the decision-rule used to trigger a case flag; 

• the points (i.e., statuses) in the service process to which 
flags are applied; 

• the amount of time-in-status that triggers the case flag; and 

• the type and amount of information provided by the flagging 
system. 

Before discussing these issues, a few general comments can be made. 
First, we found no evidence to suggest that flagging times were adjusted 
by states in response to variations in client type. Thus, agencies did 
not establish longer flagging times for severely or multiply disabled 
clients, nor did they distinguish among different types of disabilities 
when setting flagging times. Although this is somewhat surprising given 
the intuitive appeal of differential flagging times for different client 
types, it may be due to the need for a more sophisticated computer program 
to make the adjustments. 

Second, in the 26 agencies with case flagging systems, all but one 
request monthly lists of clients in-process; the other agency requests 
this information on a quarterly basis. These lists provide either a full 
accounting for all in-service cases (a "master list") with the delayed 
cases being highlighted, or only provide a list of those delayed (as 
appropriate to the agency's flagging time). In most agencies the pro- 
cedure follows a "top down 11 process; that is, the central or district 
office provides the list to the counselors and supervisors. However, in 
a few cases we found a "bottom up" approach, wherein the counselors would 
produce reports on their clients, for the information purposes of the 
agency's management personnel. 
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As would be expected, all of the case flagging systems are based on 
time measurements within particular statuses. However, the decision rules 
used to flag cases vary across agencies. The following variety of decision 
rules were found across our sample of responding agencies: 

• Flagging occurs after a case has remained in a particular 
status past the established maximum time. In such cases, 
review must occur and, barring extenuating circumstances, 
counselors must move the case on to the next appropriate 
status (including closure). 

e Flagging occurs after some "desirable 11 period of time-in- 
status, rather than after an established maximum time is 
exceeded. In such cases, the flag serves an "advisory 11 func- 
tion, rather than serving to note cases where action is man- 
datory. 

• Flagging occurs because no counselor-client contact has 
occurred within some established period of time. This time- 
period varies as appropriate to the client f s current status; 
for example, longer time periods are allowed between contacts 
for clients in training (status 18) than for clients in plan 
development (status 10). 

• Flagging occurs based on need for review. For example, if a 
state reviews all cases in status 02 after 90 days, then 
flags will point out those cases that either are approach- 
ing or have recently passed the 90-day period. Delay prob- 
lems, if any, will be discovered during review. i 

The next dimensions on which different states 1 flagging systems vary 
pertain to the statuses for which flagging times have been established, 
and the exact time norms established for those statuses. Table 1 summar- 
izes the existing flagging systems. The table shows the apparent special 
scrutiny given to "non-service" statuses in the flagging process. Whereas 
the three core service statuses (14, 16, and 18) are flagged by 11 to 15 
of the agencies, the other statuses are each flagged by a minimum of 19 
agencies. 

As would be expected, more time is allowed to elapse in the core ser- 
vice statuses, in relation to the other statuses, before flagging occurs. 
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Table 1 

Summary of Case Flagging Parameters: Statuses 
Flagged and Summary Statistics for Statuses 



Status 


N* 


Average 
of Responses 
(Months) 


Range 
of Responses 
(Months) 


Modal Response 
(Months) 


00 


22 


3.2 


2-6 


3 


02 


24 


4.6 


2-6 


6 


06 


20 


15.4 


2-19 


18 


10 


24 


4.6 


2-10 


6 


12 


21 


4.3 


2-12 


3 


14 


15 


13.3 


4-48 


12 


16 


14 


13.4 


4-48 


12 


18 


11 


39.1 


1-99 


48 


20 


19 


4.2 


2-9 


3 


22 


22 


3.8 


2-6.5 


3 


24 


20 


5.7 


2-12 


6 



♦Number of agencies which have established flagging times 
for the particular status 
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Thus, for core service statuses, the average time lapse prior to flag- 
ging ranges from 13.3 months (status 14) to 39.1 months (status 18). 
In contrast, with the exception of extended evaluation (in which extensive 
services can be provided), the remaining statuses all have average flag- 
ging times falling within the range of three to six months. However, 
within any given status, agencies vary widely in the particular flagging 
times used. The narrowest range occurs for the initial eligibility deter- 
mination statuses: flagging times for both status 00 and 02 ranged from 
a minimum of two months to a maximum of six months. In contrast, the 
planning statuses (10 and 12) showed a much wider range of flagging times 
(from two to ten months) as did two of the "post-service 11 statuses (statuses 
2C and 24). Interestingly enough, the flagging times for status 22 (in 
employment) tend to be set at a higher level than one would expect. That 
is, the law specifies that clients shall be in employment, for at least 
60 days prior to closure into status 26. One would expect agencies to 
flag cases after that minimum time had elapsed — approximately two months - 
in order to enhance their 26-closure rate. However, only three of the 
26 agencies flag at two months. Finally, the widest range of flagging 
time was found for the core service statuses: the range was from four 
months to four years for statuses 14 and 16, and from one month to over 
eight years for status 18 (training). 

The last dimension over which flagging systems vary is that of the 
information provided by the systems. The basic information is, of course, 
the amount of time elapsed since the last "milestone 11 (i.e., status change 
or client-counselor contact). For the majority of our responding agencies, 
this is the only information provided for monitoring timeliness. However, 
a few agencies do provide supplemental information for use by counselors 
and supervisors. For example, some agencies use a multi-level flagging 
system; cases in status 02 get one star at 90 days, two stars at 120 days, 
and three at 150 days. Usually, these levels denote the following: the 
first level indicates "best practice," in the sense that that time period 
is associated with successful outcomes. Thus, this may be the average 
time- in- status experience by 26s, or the period within which 75% (or 90%, 
etc.) of their 26 closures passed through the particular status. The 
second level denotes the. maximum "desirable" time by which the case should 
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move to a new status. If this time is exceeded, cases should undergo 
review. Finally, the third level denotes the time by which (barring 
unusual circumstances) some movement must occur for the case. 

A second type of system found in our survey provides special sum- 
mary reports for particular groups. For example, one state provides coun- 
selors with reports on "referral and applicant backlogs" and on "unusually 
active or inactive clients," in addition to the basic client flow report. 

•Finally, some agencies have incorporated the timeliness concept into 
their goal setting and performance assessment systems. For example, some 
agencies have set policies regarding the maximum acceptable number or 
proportion of clients flagged; e.g., "less than 10% of our 02s will have 
flags at any one time." This concept has been extended even further in a 
few agencies, to incorporate a measure of progress over time. Thus, such 
agencies set goals for a certain percentage reduction over time in the 
number of cases flagged, and/or in the average time-in-status experienced 
by clients. 

This concludes our review of existing state agency case flagging sys- 
tems. We turn next to a presentation of the findings of our analysis of 
the standards pretest data. 
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IV. ANALYSIS OF SAMPLE TIMELINESS DATA 
FROM FOUR STATES 

Reviewers from the staff of each of the Model Evaluation Units in 
four states -- Delaware, Michigan, Oregon, and Mississippi -- assessed 
the timeliness of movement through the VR process for a sample of closed 
cases. Separate assessments were made for the referral and acceptance 
stage (statuses 00-08/10), for the plan development stage (statuses 10-12), 
and for the service process (statuses 12-26/28/30) . These assessments were 
judgmental, in the sense that no specific quantitative standards (i.e., 
times-in-status) were applied to determine timeliness. Rather, the assess- 
ments were based on judgments regarding the appropriateness of the speed 
of case movement, given the various circumstances impacting a particular 
client's casef low. Reviewers needed to be thoroughly familiar with the 
case circumstances before attempting to assess timeliness. Familiarity 
with each case was gained by completing the CRS, the R-300 Verification 
instrument, or by conducting an independent review of the case file. 

All of the reviewers were state agency staff trained in conducting 
the timeliness assessment. Training for the timeliness iassessments took 
place in each state either as an adjunct to the Case Review procedure 
(Oregon and Pennsylvania), to the R-300 Verification procedure (Delaware, 
Mississippi, and Virginia), or as an independent activity (Michigan). 

Interrater reliability was analyzed for each state by SDSU. The con- 
sistency ratios for the timeliness assessment are presented in Table 2. 
A consistency ratio of .75 was considered to be adequate reliability. 
Pennsylvania, Virginia, and Mississippi have very high consistency ratios 
(all above .80) on all three timeliness assessments. Michigan and Oregon 
each have some very high consistency ratios and one that is just adequate. 
The lack of high consistency ratios across all states is indicative of the 
judgmental nature of the timeliness assessments. It is clear that in some 
cases, reviewers did not agree on what constitute timely case handling. 
In order for the validity and reliability of these measures to be improved, 
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Table 2 

Consistency Ratios for Timeliness Assessments 





DE 


MI 


MS 


OR 


PA 


VA 


Consistency Ratio 
for Acceptance 
(00-08/10) 


.50 


.96 


.98 


.87 


.81 


.95 


Consistency Ratio 
for Plan Development 
(10-12) 


.75 


.75 


.93 


.92 


.82 


1.00 


Consistency Ratio 
for Service Provision 
to Closure 
(12-26/28/30 


.50 


.79 


.86 


.70 


.83 


.88 
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reviewers will need additional training, and the establishment of criteria 
for determining timeliness. Even with some measurement problems, an analy- 
sis of these data can still provide RSA with useful ideas and information 
on what factors need to be examined in deciding how timely case movement 
can be ensured. 

In the rest of this section, we present our findings from analysis 
of data from clients in Delaware, Michigan, Oregon, and Mississippi for 
whom outcome data is available in the pretest. The analysis includes an 
examination of the timeliness assessments and how they relate to outcomes, 
time-in-process, client characteristics, and process characteristics. 

DESCRIPTION OF TIMELINESS 

Timeliness of Acceptance 

Timeliness assessments for the acceptance phase, 00-08/10, are pre- 
sented in Table 3. The reasons why cases were rated as untimely are also 

presented in this table. Overall, 78% of the cases were rated as timely, 

2 

indicating that most cases go through this phase without undue delay. 

The assessment of timeliness does vary greatly by state. Delaware 
rated only 5% of its cases as timely, whereas all (100%) of Oregon's cases 
were considered timely. The reasons why there is such a large difference 
among states is unclear. It is very possible that cases in Oregon and 
Mississippi do not experience as much undue delay as cases in Delaware and 
Michigan. On the other hand, the differences between states may be reflec- 
tive of different criteria used to assess timeliness. The timeliness 
assessments for each state were made by reviewers from that state. And, 
although interrater reliability within a state was examined, state time- 
liness assessments do not examine the differential judgments of timeliness 
across states. An examination of rater by rater variations in timeliness 
judgments could be performed by modeling the relationship between the 

*In Pennsylvania and Virginia, the timeliness data were not collected 
on clients with comparable outcome data. 

2 

The pretest sample, however, only includes 26s and 28s. The time- 
liness of 08s was not assessed. Whether or not 08s experience the same 
level of timeliness is unknown. 
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Table 3 

Assessment of Timeliness 
for Acceptance (Status 00-08/10) 





Delaware 


Michigan 


Oregon 


Missis- 
sippi 


Total 




5 7 9 6 


11% 

/ £m 0 


100% 

X \J\J 0 


89% 


78% 


00 


(49) 


(95) 


(34) 


(83) 


(261) 


Reasons for Being 
Rated Untimely: 












No response by client 


10% 


11% 




0% 


9% 


Client indecision 


10 ? 6 


4% 




44% 


13% 


Client unavailability 


15% 


4% 




0 


7% 


No contact by counselor 


15% 


22% 




22% 


20% 


No action by counselor 


35% 


48% 




33% 


41% 


No counselor assigned 


5% 


0 


- - 


0 


2% 


Administrative delays 




U 




U 


14% 


Lack of resources 


0 


0 




11% 


2% 


Delay in receiving 
reports 


10% 


26% 




22% 


20% 


Interagency delays 


30% 


0 




0 


11% 


Lack of placement 
opportunities 


0 


0 




0 


0 


Other 


30% 


22% 




0 


21% 


(N) 


(20) 


(27) 




(9) 


(56) 
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judgment itself, the time facts (days in status), and the demographic 
facts of particular cases. To determine if different criteria were used 
in making assessments across states, either an across-state reliability 
check should be done, or a single team of reviewers should assess the 
cases in all states. 

The most common reason for delay was "no action by counselor, 11 This 
reason was cited for 41% of the untimely cases, "No contact by counselor" 
(20%), "delay in receiving reports" (20%) , and "other" (21%) were the next 
most common reasons for delay. Every state cites "no action by counselor" 
as a common reason for delay during the acceptance phase. Otherwise, the 
states varied quite a bit. In Delaware, "administrative delays" was the 
most often mentioned cause of delay. This reason, however, was not men- 
tioned in either Mississippi or Michigan, "Client indecision" was the 
most common reason in Mississippi, but was rarely mentioned in the other 
two states. These differences may reflect the differences in state prac- 
tices and experiences. They may also be reflective of differences in how 
the states completed the assessment forms. 

Timeliness of Plan Development 

Only 11% of all cases were rated as not timely for the plan develop- 
ment phase (see Table 4), Hence, undue delay during this phase does not 
appear to be a problem. Although all states reported a high percentage 
of cases as timely, there is a difference between states. Delaware rated 
the fewest percentage as timely while Oregon and Mississippi rated the 
most as timely. 

"No contact by counselor" and "no action by counselor" were the most 
common reasons for delay. They were each frequently mentioned by both 
Delaware and Michigan. "No response by client" was also a common reason 
for delay in Michigan. Conclusions from these findings, however, cannot 
really be made given that so few cases (26) were untimely. 

Timeliness of Service Provision 

Of the three assessments of timeliness, service provision has the 
lowest rating. Sixty-four percent of the cases were considered timely 
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Table 4 

Assessment of Timeliness 
for Plan Development (Status 10-12) 





Delaware 


Michigan 


Oregon* 


Missis- 
sippi 


Total 


% Rated as Timely 
CN) 


77% 
(49) 


84% 
(95) 


94% 
(34) 


100% 
(83) 


89% 
(261) 


1 " — 1 

1 

Reasons for Being 
Rated Untimely: 












No response by client 


18% 


38% 






27% 


Client indecision 


9% 


7% 






8% 


Client unavailability 


9% 


15% 


_ — 




19% 


No contact by counselor 


45% 


54% 





— 


46% 


No action by counselor 


36% 


31% 





— 


31% 


No counselor assigned 


0 


15% 






8% 


Administrative delays 


0 


7% 






4% 


Lack of resources 


0 


15% 




— 


8% 


Delay in receiving 
reports 

i 


79% 


15% 






12% 


Interagency delays 


0 


7% 






4% 


Lack of placement 
opportunities 


0 


0 






0 


Other 


9% 


15% 






15% 


CN) 


(11) 


(13) 






(26) 



*Since only two cases were untimely in Oregon, their reasons for 
untimeliness were deleted from this table. They are included in 
the overall reasons for being untimely. 
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(see Table 5). The variation between states is striking, Delaware rated 
37% as timely, while Mississippi rated 84% as timely. The difference 
between states in the timeliness of service provision is the same as with 
the acceptance and plan delivery phases, Delaware was the lowest, while 
Oregon and Mississippi were the highest. Again, whether this is due to 
actual differences in case handling or to different rating criteria is 
not known. 

"Client unavailability 11 and "no contact by counselor" were the major 
reasons cited for cases being untimely. "No response by client" and "no 
action by counselor" were also frequently mentioned. The reasons for delay 
also varied by state. For example, "client indecision" was often cited 
in Oregon and Mississippi, but rarely in Delaware and Michigan. 

In summary, although the states differ in the specific reasons cited, 
delays caused by both clients and counselors were common in all states. 
Client-oriented reasons (e.g., indecision, unavailability) were frequently 
mentioned as reasons for delay during service provision. These reasons 
were much less common for the acceptance and plan development phases. 
Counselor-oriented reasons for delay (e.g., no contact, no action) were 
frequently cited for each of the three phases and in all of the states. 
This suggests that some type of case flagging mechanism that would alert 
counselors of undue delay could be very useful. Surprisingly, program- 
matic reasons (e.g., administrative, lack of resources) were rarely seen 
as causing delays. Only during the acceptance phase were these reasons 
common • 

Our data show a strong association between outcome and the subjective 
assessments of t^eliTlg§§ t Tabl§ 6 iflPWS the relationship between closure 
status and assessed timeliness at the acceptance, plan development, and 
service provision phases. Seventy percent of the clients rated timely at 
acceptance were closed 26, while only 53% of those who were untimely were 
closed 26. The relationship is less strong at the plan development phase, 
with 68% of those assessed timely being closed 26 as compared to 54% of 
those not timely. Timeliness during service provision is very strongly 
related to outcome. Eighty percent of those rated as timely during this 
phase were closed 26, while only 43% of those not timely were closed 26. 
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Table 5 

Assessment of Timeliness 
for Service Provision (Status 12-26/28/30) 





Delaware 


Michigan 


Oregon 


Missis- 
sippi 


Total 


% Rated as Timely 
CN) 


37% 
14 8) 


57% 
(90) 


71% 
r xa "\ 

(..54J 


84% 


64% 


Reasons for Being 
Rated Untimely: 












No response by client 


23% 


37% 


40% 


8% 


29% 


Client indecision 


10% 


13% 


40% 


38% 


19% 


Client unavailability 


23% 


40% 


70% 


31% 


36% 


No contact by counselor 


30% 


47% 


20% 


23% 


35% 


No action by counselor 


23% 


34% 


10% 


15% 


25% 


No counselor assigned 


7% 


24% 


0 


0 


12% 


Administrative delays 


20% 


8% 


0 


15% 


12% 


Lack of resources 




10. 

3-6 


o 


0 


•70, 


Delay in receiving 
reports 


3% 


5% 


0 


8% 


4% 


Interagency delays 


0 


3% 


0 


0 


1% 


Lack of placement 
opportunities 


0 


5%' 


0 


0 


2% 


Other 


37% 


11% 


0 


31-% 


21% 


CN) 


(30) 


(38) 


(10) 


(13) 


(91) 
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Table 6 

Relationship between Timeliness Assessments 
and Outcome 





Outcome J 




zo Liosures 




At Acceptance (00-08/10) 






Timely 


70% 


(204) 


Not Timely 


53% 


(57) 


Chi Squared Significance 


.02 




At Plan Development (10-12) 






Timely 


68% 


(233) 


Not Timely 


54% 


(28) 


Chi Squared Significance 


.19 




At Service Provision (12-26/28/30) 






Timely 


80% 


(163) 


Not Timely 


43% 


(92) 


Chi Squared Significance 


.00 
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Although these findings show a strong correlation between timeli- 
ness and outcomes, the correlation may be artificially high. In rating 
cases as timely or not, reviewers were aware of the clients 1 closure 
status. For some cases, this knoweldge most likely had an impact on their 
assessments. For cases that had clearly experienced or had clearly not 
experienced undue delay, assessing their timeliness was a straightforward 
task. But for those cases where it was difficult to assess whether or 
not an undue delay had occurred, a reviewer may have been influenced by 
the closure status. For example, a case that experienced some delay but 
was closed 26 might have been rated as timely, the reasoning being that, 
although there was some delay,, it was not enough to cause a problem for 
the case in the long run. A similar case that was closed 28, however, 
might easily have been rated as not timely. Here, the reviewer might 
have seen the delay as one of the problems in preventing a positive closure. 
The extent to which closure status influenced the timeliness assessments 
is unknown. In practice, timeliness assessments would be made prior to 
case closure. 

The observed association between timeliness and positive outcomes 
does not provide conclusive proof of a direct causal relationship in which 
timely case handling facilitates positive outcomes. Several alternate 
explanations are also possible. For example, mQre severe cases that may 
take a longer time to be served in each status may also be less likely to 
have positive outcomes. Alternatively , cases that, for whatever reason, 
are a poor risk for success may be given less priority by the counselor. 

Relationship Between Timeliness and Time in Process 

Most previous research has used time in process to measure timeliness 
of case handling. This approach has been justifiably criticized because 
of its lack of sensitivity to the legitimate differences in individual 
cases. As cases remain in the VR process for longer periods of time, 
they are less likely to be considered timely. However, there are some 
cases that have gone through the VR system quite rapidly and are still 
rated as untimely. Likewise, some cases that have been in the system for 
-extended periods of time are considered to he timely. In this analysis, 
we have used judgmental assessments of tixCliness that take into account 
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the individual needs of clients. In comparing these assessments with 
time in process, we find that there is a strong relationship between the 
two . 

Figure 3 illustrates the relationship between the timeliness assess- 
ments and time in process for the acceptance phase in each of the states. 
Except in Oregon, where all of the cases were rated as timely, as the 
number of months in statuses 00-06 increases, the percent timely decreases. 
The relationship between timeliness and time in process, however, is dif- 
ferent for the different states. In Delaware, a high percentage (87%) of 
those in the acceptance phase for 0-3 months are rated as timely. There 
is a sharp drop, however, for those who have been in this phase for four 
or more months. Only 28% of these clients were rated as timely. All of 
the clients in Mississippi who remain in statuses 00-06 for zero to three 
months are rated timely. Although fewer clients (69%) who are in the 
phase four or more months are considered timely, the drop in Mississippi 
is not nearly as sharp as that in Oregon. Whereas the patterns found in 
Oregon and Mississippi are similar, Michigan is quite different. Michigan 
is the only state that shows a decrease in the percent rated as timely 
between zero and one month and two to three months. All of Michigan's 
clients who went through statuses 00-06 in a month or less were considered 
timely. For those who remained in this phase for two to three months, 
the percent rated timely dropped to 84%. An even larger decrease occurred 
for those who remained in this phase for four or more months, with 42% of 
them being rated as timely. 

The relationship between timeliness and time in process during the 
plan development and service provision phases is presented in Figure 4. 
Again, there is a strong inverse relationship. As time in process in- 
creases, the percent rated as timely decreases. The differences between 
states for these phases is even more striking than that found in the 
acceptance phase. In all states, a very high percentage of cases that 
complete statuses 10-24 in six months or less are rated as timely. In 
Delaware and Michigan, there is a large drop in the percent timely for 
those in service for 7-12 months. The percent rated as timely decreased 
39% in Delaware and 35% in Michigan between these two time periods. By 

"^Since R-300 data does not provide data for time spent in the plan 
development and service provision phases separately, the timeliness assess 
ments have been combined. Cases were considered timely only if they were 
q rated as timely for both phases. 
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Figure 3 

Relationship Between Timeliness Assessments and Time In Process 
by State for the Acceptance Phase 
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Figure k 

Relationship Between Assessments 
and Time in Process by State 
for the Plan Development and Service Provision Phases 
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contrast, in Oregon and Mississippi, there is virtually no difference 
between the percent rated as timely in these two periods. 

In all four states, a smaller percentage of clients in service for 
13-24 months are rated as timely than those in service for 7-12 months. 
In Delaware and Oregon, the percentage drop is quite striking (26% and 
36% respectively), while in Mississippi it is only a moderate drop (13%). 

Of the four, Mississippi alone showed a noticeable difference in the 
percent rated timely in the 13-24 month category as compared to the over- 
two-year category (78% of the cases in service 13-24 months compared to 
50% of the cases in service for over two years) . Delaware shows a minimal 
decrease while Michigan and Oregon show minimal increases in the percentages 
rated as timely between these two periods. 

Looking at the states individually, the following patterns in the 
relationship between timeliness and time in process for the plan develop- 
ment and service provision phases emerge: 

• Delaware : A large drop in the percentage rated as timely at 
six months and 12 months. Virtually no change at 24 months. 

• Michigan : A large drop at six months and a moderate drop at 
12 months. Virtually no change at 24 months. 

• Oregon : A large drop at 12 months. Virtually no change at 
six months and at 24 months. 

• Mississippi : A moderate drop at 12 months and a large drop 
at 24 months. Virtually no change at six months. 

It is clear that the critical points, for determining timeliness 
during both this phase and the acceptance phase differ by state. Why 
these differences occur, however, is not obvious. They may be a result 
of different state practices. A different approach to service delivery 
could mean that a particular case being served for over six months in 
Delaware or Michigan would not be timely, while the same case in Oregon 
or Mississippi would be timely. If different state practices are the 
reason why states vary, a uniform case flagging system that relied on 
time in process would not be appropriate. Such a system would not be 
sensitive to the different state practices and hence would not be useful 
in eliminating undue delay. 
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The differences between states in the relationship between timeliness 
and time in process may also be due to different definitions of timeliness. 
As stated earlier, the study did not include measurement of interrater 
reliability across states. Hence, it is not known if states used the 
same criteria in assessing timeliness. It would not be surprising if 
reviewers in different states used different time cut-off points when mak- 
ing their assessments of timr liness, and the variations should be identi- 
fied across all states. Before any standard mechanism for case-flagging 
is implemented, a better understanding of the reasons for differences in 
timeliness across states must be obtained. From our findings, it is un- 
clear as to whether the differences between states are real or artificial. 
If the differences are real, a case-flagging system should be tailored to 
each state. If the differences are simply an artifact of how the data were 
collected, then a uniform case-flagging system may be appropriate. However, 
states may use their own timeliness procedures to implement the approach 
BPA recommends later in these pages. 

FACTORS AFFECTING TIMELINESS 

It has been suggested that in order for a case-flagging system to 
be useful in ensuring timely case handling, it should be sensitive to the 
needs of individual clients. Many think the ideal case-flagging system 
would set time cut-off points differently for different statuses, for 
different client types, and for different service packages. Thus, in 
our analysis we attempted to isolated specific factors that affect time- 
liness and that could be included in a case-flagging system. 

In trying to explain why cases were or were not considered timely, 
we have looked at a ser.les of variables representing both client charac- 
teristics and the clients 1 service process experience. Client character- 
istics include variables such as age, race, sex, education, and previous 
work experience. A client's specific disability, as well as the severity 
of the disability, might also show a relationship to timeliness. It may 
be that clients with certain disabilities are more likely to experience 
undue delay. Further, severely disabled clients, special program clients, 
or those clients with secondary disabilities may be more difficult to 
serve and more prone to delays. 
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In order to assess the relative impact of each of these variables on 
the two measures of timeliness (timeliness at acceptance and timeliness 
at plan development and service provision), multiple regression analysis 
was used. The results are discussed below. 

Timeliness at Acceptance 

Other than absolute time in status, the variables that showed some 
statistically significant association to lack of timeliness at acceptance 
were the following: 

o Severity of Disability : nonseverely disabled clients, who were 
7% more likely to be judged untimely than severely disabled 
clients*; 

• Type of Disability : "other 11 disabilities, who were more 
likely to be judged untimely than clients with visual/aural, 
mental/psychological, or orthopedic/amputee disabilities; and 

• Ag e : clients over 45 years of age, who were 11% more likely 
to be judged untimely than younger clients. 

The variables that were related to lack of timeliness at plan development 
and service provision included: 

• Type of Disability: clients with "other 11 disabilities were 26% 
more likely to be judged untimely than those with visual or 
aural disabilities; 

• Previous Work Experience : Clients who had some work experience 
before entering VR were 19% more likely to be judged untimely 
than those with no work experience; * : 

• Race : nonwhite clients were 19% more likely to be judged 
untimely than white clients; 

• Special Program Status : 110 clients without trust fund monies 
were 15% more likely to be judged untimely than clients who 
received SSI and SSDI. 

• Severity of Disability : non-severoly disabled clients were 9% 
more likely to be judged untimely than severely disabled clients. 

Water's expectations that severely disabled might take more time 
could explain this difference. 
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The strong relationship between race and timeliness is particularly 
puzzling. Although it is not evident why these particular factors are 
related to judgments of tinaiiiie^s, the relationships are all fairly 

strx-rr?. . This suggest z than: ,.g grts±n - client characteristics also affect 
: /ir^izness and mi be ixi^rrainziitsd in a case-flagging system. An 
™j-sis cf ho: :*sc, .ute ""rfsne ±a. status varies for different groups would 
ul: yi^i^ s\x k tijns abo. ~ r~w co adjust time in status monitoring stan- 
dards for different groups. 
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V. A MODEL FOR CASE TIMELINESS MONITORING 



As we have seen from our review of existing state timeliness systems, 
there are numerous approaches to monitoring cases for undue delay. The 
selection of a "best approach 11 is not currently possible, since our analy- 
sis shows a variety of state conditions and client characteristics that 
may affect both the time in status itself and a rater's own assessment of 
the "timeliness 11 of a particular case. While the MEU standards pretest 
data shed some light on the relationships between timeliness, time in 
process, and outcome of cases, the limited sample and measurement issues 
preclude our ability to project the findings for all state VR agencies or 
even classes of agencies. However, this investigation does indicate that 
a concern for timeliness is important both for assuring the quality of 
case services, and for providing the best chance for successful outcomes 
of these services. Based on the analysis, BPA recommends: 

• That any procedure recommended by RSA for monitoring of oper- 
ations must be reliable across states. Trained raters should 
apply the same procedures and specific criteria across states 
if the findings are to have validity and reliability. Such an 
approach argues for assessments done by RSA at the regional 
level or by a national assessment project rather than by indi- 
vidual state staff. 

• The procedures should focus on identification of untimely prac- 
tice, overall, by type of plan, by types of client, and by 
status. Tais identification should be developed from data 
analysis on improved measures of timeliness and improved docu- 
mentation of times in status on the R-300. 

• With improved measures, more analysis can be done on the rela- 
tionships between times in status and timeliness assessments, 
and outcome data from the R-300. Such analysis will lead to 
identification of norms for case flagging, and provide a com- 
ponent of the supportive evaluation or decision support system 
for RSA MIS. 
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Thus, our long-run recommendation is that timeliness measures and 
procedures must be improved and be more uniformly applied to a large 
number of cases before an empirical basis for setting timeliness norms 
is possible. We see this developmental work as becoming more possible 
through the MIS, if timeliness assessments and status times are included 
in the data elements and if the uniform assessment practices are followed 
in collecting the data. 

In the short run, we suggest the following model for improvement of 
case flagging practice and use of time-in-status standards at the state 
level. As we pointed out in the first chapter of this timeliness analysis, 
a state should not flag too many cases, because such flagging would be 
inefficient. However, flagging too few cases will possibly leave too many 
untimely cases in the system without examination. The model, therefore, 
is based on examining the number of cases being flagged with the states 1 
existing flagging standards, in relationship to an analysis of the service 
process and the overall caseload, and refining state flagging standards. 

Figure 5 shows the steps to follow in the analysis of undue delay in 
the caseload. A case is selected for review, following the sampling 
design used by the state agency for timeliness review (100% of all cases, 
or random sampling procedure sufficient to produce desired sample size). 
State timeliness standards (allowed times-in-process) will be used in this 
model. The model permits upwards or downwards adjusting of these time-in- 
process standards (increases or decreases in the maximum time allowed in 
each status) based on two additional suggested standards for the caseload 
itself: 

1. Not more than 20% of flagged cases should be ultimately found 
to be timely. If more than 20% of the flagged cases are judged 
as timely when a case is reviewed, the system may be flagging 
cases unnecessarily, and the flagging standards could be made 
less stringent (times allowed in the statuses could be in- 
creased) . 

2. Assuming the conditions in (1) above hold, at least 5% but 
not more than 10% of all cases should be flagged. If such 
excessive flagging occurs, and the flagging represents cases 
judged untimely, then there may be a problem with the service 
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Figure 5 
Model Case Flagging System 
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delivery system itself, and an analysis of the process is 
called for. If less than 5% of cases are flagged, the flag- 
ging system may be made more stringent (times allowed in 
statuses may be decreased). 

The model in Figure 5 uses both quantitative information on times- 
in-status and judgmental information on timeliness to decide about cases. 
In the first stage, a case is selected for attention. If time-in-process 
is within acceptable limits for the case, it is returned to the file. If 
time-in-process exceeds agency standards, the case is flagged and reviewed 
as to the timeliness of process. As we have pointed out, while there is 
a relationship between timeliness and time-in-process, it is not a one- 
to-one relationship, so it is possible that flagged cases will be judged 
timely . If so, they also can be returned to the file. For both these 
groups returned to the file (labeled A and B on our diagram), the number 
of such cases should be recorded. Likewise, the number of flagged cases 
judged to be untimely should be counted. (In our diagram; this is C.) 
Cases should be flagged and reviewed until the planned sample size (A + 
B + C) is achieved. 

Once the sample is complete, the system asks three questions of the 
cases. First, do the timely cases (B) exceed 20% of all flagged (B + C) 
cases? (If yes, the system may be flagging too many cases, and times 
allowed for each status could be increased.) If, however, B/ (B + C) is 
less than 20%, we ask whether less than 10% of the total caseload was 
flagged. If so, then the time- in-process standards appear in equilibrium 
for the state (not too many cases are being flagged; of the cases that are 
flagged, most of them are, indeed, untimely cases). If, on the other hand, 
more than 10% of the cases are flagged, there is a problem in the service 
process itself, since these cases have been judged as untimely and there 
are too many untimely cases for efficient monitoring and efficient opera- 
tions. This calls for an examination of the service process itself, to 
analyze the state caseload process and pinpoint timelines "sues in rela- 
tionship to client outcomes and costs. In addition, this problem may call 
for upward adjustment of the times allowed in statuses, _to flag fewer 
cases. However, once changes in the time-in-status standards have been 
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made, another iteration must.be performed so that a check can be made 
as to whether both the 20% and 10% standards are met for a given new 
standards level. 

Finally, we suggest that the state should routinely flag between 
5% and 10% of its cases, to assure that flagging standards are set low 
enough. If less than 5% of cases are f }*.jjged, the standards should be 
made more stringent (allowed time-in-status decreased) before the next 
round of review, 

Using this approach, states can adjust their time-in-status standards 
upward or downward to be more meaningful and to result in an efficient 
process that spots problematic cases without excessive monitoring. 



